1. Field of the Invention
The present invention relates to a method for reducing processing power requirements of a video decoder, and more particularly, to a digital video decoders and the reduction of computational requirements of these decoders.
2. Description of the Related Art
Digital video has become widespread in the field of consumer electronics, due in large part to the emergence of digital video standards such as MPEG1 (see xe2x80x9cCD11172xe2x80x94Coding of Moving Pictures and Associated Audio for Digital Storage Media at up to about 1.5 Mbpsxe2x80x9d by International Organisation for Standardisation, ISO MPEG Document, ISO-IEC/JTC1/SC2/WG11, 1994), MPEG2 (see xe2x80x9cIS13818 Generic coding of Moving Pictures and Associated Audioxe2x80x9d by International Organisation for Standardisation, ISO MPEG Document, ISO-IEC/JTC1/SC2/WG11, 1994), and the emerging standard MPEG4, which includes further added functionality and quality over MPEG1 and MPEG2. In MPEG1 and MPEG2 decoders, there are certain expectations regarding the decoder computational power which must be met in order to conform to the respective standard. For example, in the MPEG2 standard, Main Profile and Main Level (MP@ML) describes a set of bounds on the bitstream and associated video which all conformant MP@ML decoders must adhere to.
There are many situations where a decoder lacks sufficient computational power to decode a bitstream as it was intended. One example is on the internet where typically MPEG1 video bitstreams can be found and decoded using computers varying from high end scientific work stations to simple personal computers. When computational power is limited, one way the decoder can reduce it""s computational requirements is to simply slow down the decoder output picture rate enough to permit all decoding to be accomplished. This was a very common method used by software based MPEG1 video decoders running on personal computers. Another method is to periodically skip the decoding of some pictures. For example, bi-directionally predicted pictures (B-pictures) can be skipped, effectively reducing the decoder output picture rate by picture-dropping. MPEG4 includes the concept of video objects, which are picture sequences which may be used to describe a separable object in a scene. If a scene contains many objects, the least important objects can be dropped in favor of giving the necessary computational power to the decoding of the more important objects.
There is a growing abundance of digital video bitstreams available, and a growing number of computational requirements associated with each. Due to the wide range of platforms that can be used for video decoding, there is a need for decoders which can decode bitstreams without requiring the full computational power required to meet conformance criteria. Existing techniques such as picture display rate reduction, picture dropping or object dropping techniques reduce the computational power at the expense of decoded video picture quality and synchronisation. For example, changing the picture display rate and picture dropping can have an effect on the perceived synchronisation of audio and video, commonly referred to as lip-synch. It also has the effect of making scenes with motion appear jerky.
An object of this invention is to permit video decoders to reduce the computational requirements of decoding a bitstream without affecting the picture rate. This will permit lip-synch to be maintained while compromising visual quality. It will also permit scenes which become jerky using picture dropping techniques to appear more smooth. Similarly, it will permit more objects to be present in the scene, since the computational power can be allocated to more objects, at the expense of object quality.
An object of this invention is to permit video decoders which lack the computational power to decode a bitstream to conformance, to decode a bitstream with varying degrees of visual quality degradation. This will permit a wide variety of decoders to be made, each with varying cost/performance trade-offs. An expensive decoder with a very high computational power can be used to decode bitstreams with a much better resulting quality than an inexpensive decoder with low computational power. By using this invention, the same bitstream may be decoded by a wide range of computational power decoders, to result in varying degrees of acceptable visual quality.
An object of this invention is to reduce the cost of video decoders. By designing video decoders that are able to meet constraints such as those imposed by MPEG2 MP@ML limitations, excessive computational power overhead must be built into the decoder to guarantee real time picture decoding of worst-case (high complexity) bitstreams. In other words, the pictures are decoded on time even when the worst case bitstream is decoded. However, by using this invention, decoders can be made which gracefully degrade picture quality when their performance limitations are met.
For the purpose of solving the above described problems, the method of reducing processing power requirements of a video decoder described herein was invented. Said processing power requirements may be controlled based on a throttling amount. Said throttling amount may be based on one or more measures of the processing power required to decode one or more bitstreams, one or more measures of said decoder""s processing capabilities, or some combination of these measures.
The measures of the processing power required to decode a bitstream may comprise of an indication of the syntax elements in said bitstream, an indication of the type of processing required for said bitstream, an indication of the amount of processing required for said bitstream, or some combination of these indications.
Reduction of processing power requirements is accomplished by reducing the processing power used for one or more decoder functions by limiting said decoder function in some way.
The limiting may comprise turning off the use of one or more of the motion vectors normally used to retrieve a motion compensated reference. This may be done by selecting from a choice of motion vectors which normally are used to retrieve a pixel regions motion compensated reference predictor region, one or more motion vectors whose use is turned off.
Selection of said motion vectors may comprise of selecting the motion vectors which are of temporally longest distance, selecting the motion vectors which are least highly correlated with those motion vectors in surrounding regions of the picture region being decoded, selecting the motion vectors which are of spatially longest distance, selecting all motion vectors, or any combination of these and other selection criteria.
Another method for limiting said decoder function comprises of modifying motion vectors, possibly by limiting the precision and/or range of one or more of the decoder""s motion vectors.
Another method for limiting said decoder function comprises of reducing the number of coefficients inverse quantized and inverse DCT transformed by selectively setting coefficients to alternate values, such as zero.
Another method for limiting said decoder function comprises of reducing the amount of decoder processing used for decoding the color components of the video bitstream.
Another method for limiting said decoder function comprises of reducing the amount of processing performed on the decoder""s output video pictures.
The throttling amount serves to control which and by how much to limit said decoder functions in some way. It is possible to determine the processing power requirements from one or more bitstreams, as well as the existing processing power capabilities of the decoder, and use this information to make a decision about which and how much to throttle, or limit said decoder functions during the decoding of said bitstreams. By limiting the decoder functionality, the processing power can be reduced.
The identification of which functions to limit and in what way can be made using indications such as the syntax elements in said bitstream, and the type and amount of processing required for said bitstream. Said limiting may comprise turning off the use of one or more of the motion vectors normally used to retrieve a motion compensated reference, based on a decision made with reference to indications in the bitstream about what motion vectors exist, or some other indications from the bitstream or decoder.
The choice of what motion vectors in a picture to turn off can be further refined by selecting the motion vectors which are of temporally longest distance, selecting the motion vectors which are least highly correlated with those motion vectors in surrounding regions of the picture region being decoded, selecting the motion vectors which are of spatially longest distance, selecting all motion vectors, or any combination of these and other selection criteria.
Similarily, said limiting may comprise of limiting the precision and/or range or some other modification of one or more of the decoders motion vectors.
Another method for limiting said decoder function comprises of reducing the number of coefficients inverse quantized and inverse DCT transformed by selectively setting coefficients to alternate values which are easy to inverse quantize and inverse DCT, such as zero. The decision to limit these decoder functions may be based on information regarding said bitstream and/or said decoder.
Another method for limiting said decoder function comprises of eliminating some or all of the processing required during color component decoding.
Another method for limiting said decoder function comprises of limiting the processing applied to the decoded video pictures.